An adaptive contrast enhancement method for stereo endoscopic images combining binocular just noticeable difference model and depth information

Endoscopic image enhancement has become a very popular research field due to the success of minimally invasive interventions and the innovation of new technological treatment and diagnosis tools such as stereoscopic laparoscopes and the wireless capsule endoscopy. In spite of the important advances achieved in terms of image processing and enhancement, only a few techniques can be adapted to stereo endoscopic images. This can be explained by the specificities of the stereo endoscopic video acquisition process, the surgical tasks artifacts and the endoscopic domain characteristics (e.g., organ textures,edges, color distribution). In this paper we present a contrast enhancement method for stereo endoscopic images taking into consideration some of these specificities, namely those of the acquired stereo images i.e. the depth information, the binocular vision and the organs boundaries/textures. The idea is to enhance the image quality by a contrast enhancement process that exploits the local image activity, the depth information and the binocular just noticeable difference (BJND) model. The results of the conducted subjective experiment show that the proposed method produces stereo endoscopic images with sharper details of the underlying tissues and organs, without introducing any halo effect or overshooting. The observers reported as well a more depth feeling and less visual fatigue when perceiving the enhanced stereo endoscopic images. Introduction During the last three decades, minimally invasive surgery (MIS) has become a popular diagnostic and treatment tool widely used in the clinical routine. While conventional open surgery relies on making large incisions in the skin and separating the underlying tissues to get a direct access to the surgical target, MIS is performed through small incisions (usually between 0.5 and 1.5 cm) to reduce the surgical trauma and morbidity. The abdomen is insufflated with a specific dose of gas in order to create a working volume through which surgical instruments can be inserted via ports. Since direct viewing of the surgical scene is not possible, an endoscopic camera assists the surgeon’s navigation by providing views of the anatomical structures and the surgical instruments. One of the main challenges facing the surgeons during the laparoscopic chirurgical training is to adapt their tasks to a two dimensional (2D) flat view of the surgical field. This lack of depth perception in addition to the loss of tactile feedback, implies a significant sensory loss for the surgeons and can affect their performance. Therefore 3D laparoscopic visual systems such as the Da Vinci Surgical System [17], the EndoSite 3Di Digital Vision System [18] and stereoscopic laparoscopes have been recently developed to address this need. The convergence to 3D visual endoscopic systems introduced, however, new issues related to image quality. Additionally, applying conventional 2D enhancement techniques on stereo endoscopic images does not give necessarily the best results as it does not account for the inherent dependencies between the perceived stereo image quality and the two views. This difficulty to adapt conventional 2D enhancement methods for stereo images may be explained by two main reasons. First, the human visual system (HVS) does not perceive the left and right images independently. The slightly different views captured by each eye are monocularly processed than fused by the visual cortex taking into account many complex binocular vision features such as the binocular rivalry and suppression depending on how much different the images are. Therefore, a depth sensitive enhancement approach exploiting a cross view processing could be a more appropriate approach to enhancing stereo 3D images. Second, most of the image enhancement methods are not adapted to the particular characteristics of the endoscopic domain (moist homogenous tissues, dynamic illumination conditions, non-rigid deformation due to the patient and surgeon motion, specular reflections), the specificities of the endoscopic video acquisition process and the surgical task artifacts (smoke, lens fogging and blood pools). Endoscopic image enhancement aims either to improve the visual video quality for the surgeons or to ameliorate the input of subsequent post processing tasks such as feature extraction for 3D organ reconstruction and registration. One of the main challenges for MIS is to determine the intra-operative morphology of the surgical field. Such information is prerequisite to the registration of the patient-specific data and to the navigation capacity providing the surgeon an efficient control of robotic-assisted surgical systems. The characteristics of the endoscopic environment including dark areas (up to 40% of the special image resolution in some cases) and different acquisition and surgical artifacts makes feature extraction from stereo endoscopic images a very challenging task, which can influence the accuracy of 3D organ reconstruction and registration tasks. Among the image processing methods that can address this problem, a proper contrast enhancement technique can improve the endoscopic image quality and the depth feeling. Indeed, it has been demonstrated in [13] that performing a sharpness enhancement on the stereo image views increases the depth perception. Based on the subjective experiment results, the authors proposed an adaptive sharpness enhancement algorithm taking into account the depth perception of the HVS. In [4], Walid et al. improve the stereo image contrast with an algorithm combining the local edge information and the depth level of each object of the scene obtained by segmenting the disparity map. An unsharp masking technique is used in [8] to enhance images containing depth information by darkening the background objects. The aforementioned methods, however, neglect the inter-view differences between right and left luminance components, which can produce visual fatigue and eyestrain for the observer. This question has been addressed by [6], in which the authors propose a sharpness enhancement technique for stereo images using the binocular just noticeable difference model (BJND) [16]. In this paper, we propose an adaptive contrast enhancement method for stereo endoscopic images combining depth information and BJND visibility thresholds. The contrast is improved combining edginess information, depth data and the local image activity to adapt the enhancement in each region (homogenous or boundary region). The BJND is then used to control the overall inter-view enhancement and avoid any noticeable difference that can trigger eyestrain or visual fatigue. The reminder of this paper is organized as follows. Section 2 describes the contrast enhancement method based on local edge detection [1]. Section 3 presents an overview of the BJND model as derived in [16]. The proposed contrast method for stereo endoscopic images is introduced in Section 4. Section 5 describes the experimental settings and discusses the results. Finally, Section 6 conclude the paper. Edge-based contast enhancement (EBCE) In this section we present an overview of the contrast enhancement based on local edge detection technique [1], which accounts for contour detection perceptual features of the HVS by combining Gordon’s method [3] and the theory of contour detection [9]. Given a pixel P at spatial coordinates (i, j) and its gray-level intensity Ii, j, the local contrast is defined as follows: Ci, j = | Ii, j−Ei, j | Ii, j +Ei, j (1) where Ei, j is an estimate of the mean edge gray-level computed by averaging the weighted gray-level intensities within a window wi, j centered at (i, j) and computed as follows: Ei, j = ∑(m,n)∈wi, j Im,n ·Φ(δm,n) ∑(m,n)∈wi, j Φ(δm,n) (2) where δm,n represents the edge value and Φ is an increasing function. The improved contrast C′ i, j can be generated by simply applying a function f to the local contrast Ci, j, satistying the following conditions: { f : [0,1]→ [0,1] Ci, j 7−→ f (Ci, j) =C′ i, j ≥Ci, j (3) The output intensity is computed as follows: I ′ i, j = Ei, j · 1−C′ i, j 1+C′ i, j i f Ii, j ≤ Ei, j Ei, j · 1+C′ i, j 1−C′ i, j otherwise (4) In [1], the authors demonstrated the efficiency and noiserobustness of this low complexity algorithm in sharpening the edges and the micro-edges (e.g., the veins) of 2D images and discriminating objects according to their boundaries. This wellknown method improves also the gray-level distribution and facilitates the detection and the extraction of relevant information such as feature points. Such data is crucial in performing a 3D organ reconstruction for the navigation and the surgery planning. Overview of the BJND The BJND model measures the minimal noise/distortion in one stereoscopic view evoking noticeable perceptual difference when combined with the other view in the binocular vision process. Based on psychophysical experiments, the authors [16] investigated the visual sensitivity to contrast masking effect, the binocular combination of noise and the luminance masking effect for stereo images. In this section, we give an overview of the BJND model and the derivation of its formula as described in [16]. Given the left and the right images, the BJND map of the left view, (i.e., BJNDl) is defined as follows: BJNDl(i, j,d) = f (bgr(i+d, j),ehr(i+d, j),nar(i+d, j)) = AC(bgr(i+d, j),ehr(i+d, j)) × ( 1− ( nar(i+d, j) AC(bgr(i+d, j),ehr(i+d, j)) )λ) 1 λ (5) where i and j refers to the spatial pixel coordinates, d is the disparity value corresponding to the point (i, j) and na is the noise amplitude 0≤ nar ≤ AC. The parameter λ controls the impact of the right-view noise and it is set experimentally to 1.25. We can notice that the BJND left is dependent on the background luminance intensity (bg), the edge high (eh) and the noise amplitude (na) of the right image. The inter-view pixel correspondence data is crucial for processing stereo content and generating the BNJD map. Such data can be provided by the ground-truth disparity information, which is not often available for real time constraints such as in our 3D endoscopic application domain. Therefore, performing a correspondence matching step (i.e., stereo matching) is important to obtain the disparity map. Stereo matching has become a very active research field in the last decade because of the difficulty and the complexity of this task for both image processing and computer vision. Since the aim of this paper is not to study nor address the stereo matching problematic, we adopt a recently proposed disparity map estimation algorithm of [7]. A more detailed review and taxonomy of stereo-matching techniques is given by [12] and a ranking of the top performing algorithms in terms of comlexity and accuracy is provided by the Middleburry website [11]. Note that if the right view is noise-free, the BJNDl is reduced to the expression of AC, which is defined by AC(bg,eh) = AC,limit(bgr(i+d, j))+K(bgr(i+d, j)) · ehr(i+d, j) (6) The background luminance component (bg) can be obtained by computing the average of a 5× 5 sliding window centered in the spatial active-pixel location (i, j), and the edge high (eh) is given by: eh(i, j) = √ E2 H(i, j)+E 2 V (i, j) (7)